Pandas Profiling Report

Dataset statistics

Number of variables	11
Number of observations	39637
Missing cells	0
Missing cells (%)	0.0%
Duplicate rows	10
Duplicate rows (%)	< 0.1%
Total size in memory	4.6 MiB
Average record size in memory	122.7 B

Variable types

Numeric	9
Categorical	2

Alerts

Dataset has 10 (< 0.1%) duplicate rows	Duplicates
`salinity` is highly overall correlated with `ORP` and 6 other fields	High correlation
`turbidity` is highly overall correlated with `Pressure in` and 1 other fields	High correlation
`ORP` is highly overall correlated with `salinity` and 6 other fields	High correlation
`TDS` is highly overall correlated with `salinity` and 6 other fields	High correlation
`Pressure in` is highly overall correlated with `salinity` and 6 other fields	High correlation
`Pressure out` is highly overall correlated with `salinity` and 6 other fields	High correlation
`Human Counter` is highly overall correlated with `ORP`	High correlation
`temperature` is highly overall correlated with `salinity` and 4 other fields	High correlation
`PH` is highly overall correlated with `salinity` and 5 other fields	High correlation
`pump current` is highly overall correlated with `salinity` and 5 other fields	High correlation
`turbidity` is highly skewed (γ1 = -55.7047875)	Skewed
`ORP` is highly skewed (γ1 = 20.66558326)	Skewed
`Pressure in` is highly skewed (γ1 = -58.60490481)	Skewed
`Pressure out` is highly skewed (γ1 = 47.0549401)	Skewed
`pump current` is highly skewed (γ1 = 60.00157855)	Skewed
`pump current` has 37477 (94.6%) zeros	Zeros
`Human Counter` has 8333 (21.0%) zeros	Zeros

Reproduction

Analysis started	2022-12-23 21:05:02.019627
Analysis finished	2022-12-23 21:05:17.043871
Duration	15.02 seconds
Software version	pandas-profiling vv3.5.0
Download configuration	config.json

salinity
Real number (ℝ)

Distinct	1642
Distinct (%)	4.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%
Mean	277.10764

Minimum	0
Maximum	557.575
Zeros	2
Zeros (%)	< 0.1%
Negative	0
Negative (%)	0.0%
Memory size	619.3 KiB

Quantile statistics

Minimum	0
5-th percentile	268.422
Q1	273.449
median	276.632
Q3	280.556
95-th percentile	283.233
Maximum	557.575
Range	557.575
Interquartile range (IQR)	7.107

Descriptive statistics

Standard deviation	8.0206101
Coefficient of variation (CV)	0.028944024
Kurtosis	494.22867
Mean	277.10764
Median Absolute Deviation (MAD)	3.599
Skewness	11.31336
Sum	10983715
Variance	64.330187
Monotonicity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
274.281	142	0.4%
280.773	136	0.3%
274.444	134	0.3%
274.389	133	0.3%
274.335	132	0.3%
274.317	129	0.3%
280.719	119	0.3%
274.353	119	0.3%
274.462	119	0.3%
280.755	118	0.3%
Other values (1632)	38356	96.8%

Minimum 10 values
Maximum 10 values

Value	Count	Frequency (%)
0	2	< 0.1%
259.85	1	< 0.1%
264.552	1	< 0.1%
264.57	3	< 0.1%
264.588	5	< 0.1%
264.606	7	< 0.1%
264.624	24	0.1%
264.642	16	< 0.1%
264.66	29	0.1%
264.678	38	0.1%

Value	Count	Frequency (%)
557.575	11	< 0.1%
335.063	1	< 0.1%
335.009	1	< 0.1%
334.882	1	< 0.1%
334.792	1	< 0.1%
334.683	1	< 0.1%
334.629	5	< 0.1%
334.484	1	< 0.1%
334.376	1	< 0.1%
334.34	1	< 0.1%

turbidity
Real number (ℝ)

HIGH CORRELATION
SKEWED

Distinct	712
Distinct (%)	1.8%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%
Mean	23.407751

Minimum	-4375.26
Maximum	46.9456
Zeros	2
Zeros (%)	< 0.1%
Negative	2837
Negative (%)	7.2%
Memory size	619.3 KiB

Quantile statistics

Minimum	-4375.26
5-th percentile	-13.74764
Q1	25.0007
median	28.4492
Q3	33.4651
95-th percentile	36.6001
Maximum	46.9456
Range	4422.2056
Interquartile range (IQR)	8.4644

Descriptive statistics

Standard deviation	75.140826
Coefficient of variation (CV)	3.210083
Kurtosis	3256.5161
Mean	23.407751
Median Absolute Deviation (MAD)	4.7024
Skewness	-55.704788
Sum	927813.04
Variance	5646.1437
Monotonicity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
33.4651	1108	2.8%
33.3083	823	2.1%
24.6873	799	2.0%
33.6218	761	1.9%
25.9412	714	1.8%
33.1516	714	1.8%
33.7786	684	1.7%
34.7192	674	1.7%
24.844	619	1.6%
25.6277	601	1.5%
Other values (702)	32140	81.1%

Minimum 10 values
Maximum 10 values

Value	Count	Frequency (%)
-4375.26	11	< 0.1%
-66.0706	1	< 0.1%
-65.7571	1	< 0.1%
-65.6003	2	< 0.1%
-65.2869	2	< 0.1%
-65.1301	2	< 0.1%
-64.9734	1	< 0.1%
-64.8167	2	< 0.1%
-64.6599	1	< 0.1%
-64.5032	1	< 0.1%

Value	Count	Frequency (%)
46.9456	1	< 0.1%
46.4753	2	< 0.1%
46.3186	1	< 0.1%
46.0051	1	< 0.1%
45.2214	1	< 0.1%
45.0645	3	< 0.1%
44.9077	1	< 0.1%
43.8105	1	< 0.1%
43.4971	1	< 0.1%
43.0269	1	< 0.1%

ORP
Real number (ℝ)

HIGH CORRELATION
SKEWED

Distinct	1182
Distinct (%)	3.0%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%
Mean	742.17061

Minimum	0
Maximum	3002.87
Zeros	2
Zeros (%)	< 0.1%
Negative	0
Negative (%)	0.0%
Memory size	619.3 KiB

Quantile statistics

Minimum	0
5-th percentile	658.037
Q1	740.864
median	756.96
Q3	763.289
95-th percentile	776.943
Maximum	3002.87
Range	3002.87
Interquartile range (IQR)	22.425

Descriptive statistics

Standard deviation	53.089104
Coefficient of variation (CV)	0.07153221
Kurtosis	913.96099
Mean	742.17061
Median Absolute Deviation (MAD)	12.84
Skewness	20.665583
Sum	29417416
Variance	2818.453
Monotonicity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
761.029	219	0.6%
761.209	203	0.5%
760.667	197	0.5%
760.938	194	0.5%
760.577	193	0.5%
760.848	191	0.5%
740.684	190	0.5%
761.481	187	0.5%
761.119	186	0.5%
761.39	184	0.5%
Other values (1172)	37693	95.1%

Minimum 10 values
Maximum 10 values

Value	Count	Frequency (%)
0	2	< 0.1%
56.4569	1	< 0.1%
121.471	1	< 0.1%
299.874	1	< 0.1%
341.468	1	< 0.1%
487.501	1	< 0.1%
490.123	1	< 0.1%
492.745	1	< 0.1%
494.011	1	< 0.1%
497.176	1	< 0.1%

Value	Count	Frequency (%)
3002.87	11	< 0.1%
781.826	1	< 0.1%
781.735	1	< 0.1%
781.012	3	< 0.1%
780.922	1	< 0.1%
780.741	6	< 0.1%
780.65	1	< 0.1%
780.56	4	< 0.1%
780.469	2	< 0.1%
780.289	4	< 0.1%

PH
Real number (ℝ)

Distinct	1468
Distinct (%)	3.7%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%
Mean	6.2382351

Minimum	-0.298727
Maximum	20.7401
Zeros	2
Zeros (%)	< 0.1%
Negative	2
Negative (%)	< 0.1%
Memory size	619.3 KiB

Quantile statistics

Minimum	-0.298727
5-th percentile	0.0974754
Q1	7.38661
median	7.4113
Q3	7.45244
95-th percentile	7.50624
Maximum	20.7401
Range	21.038827
Interquartile range (IQR)	0.06583

Descriptive statistics

Standard deviation	2.7144671
Coefficient of variation (CV)	0.43513383
Kurtosis	1.5034913
Mean	6.2382351
Median Absolute Deviation (MAD)	0.03544
Skewness	-1.757885
Sum	247264.92
Variance	7.3683317
Monotonicity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
0.0974754	1531	3.9%
0.0981084	1513	3.8%
0.0987413	1215	3.1%
0.0968425	742	1.9%
0.0993743	597	1.5%
7.45497	380	1.0%
7.40054	299	0.8%
7.39737	294	0.7%
7.3999	293	0.7%
7.4056	290	0.7%
Other values (1458)	32483	82.0%

Minimum 10 values
Maximum 10 values

Value	Count	Frequency (%)
-0.298727	2	< 0.1%
0	2	< 0.1%
0.0955765	2	< 0.1%
0.0962095	189	0.5%
0.0968425	742	1.9%
0.0974754	1531	3.9%
0.0981084	1513	3.8%
0.0987413	1215	3.1%
0.0993743	597	1.5%
0.100007	227	0.6%

Value	Count	Frequency (%)
20.7401	11	< 0.1%
13.9561	1	< 0.1%
12.1724	1	< 0.1%
11.8553	1	< 0.1%
11.5046	1	< 0.1%
10.5875	1	< 0.1%
10.5115	1	< 0.1%
9.9349	1	< 0.1%
9.85324	1	< 0.1%
9.50575	1	< 0.1%

TDS
Real number (ℝ)

Distinct	1640
Distinct (%)	4.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%
Mean	277.10733

Minimum	0
Maximum	557.575
Zeros	2
Zeros (%)	< 0.1%
Negative	0
Negative (%)	0.0%
Memory size	619.3 KiB

Quantile statistics

Minimum	0
5-th percentile	268.422
Q1	273.431
median	276.632
Q3	280.574
95-th percentile	283.233
Maximum	557.575
Range	557.575
Interquartile range (IQR)	7.143

Descriptive statistics

Standard deviation	8.0204596
Coefficient of variation (CV)	0.028943513
Kurtosis	494.26481
Mean	277.10733
Median Absolute Deviation (MAD)	3.599
Skewness	11.313758
Sum	10983703
Variance	64.327773
Monotonicity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
274.389	133	0.3%
274.281	131	0.3%
280.773	130	0.3%
274.317	129	0.3%
274.444	128	0.3%
274.462	127	0.3%
274.353	126	0.3%
274.335	125	0.3%
274.245	123	0.3%
280.719	120	0.3%
Other values (1630)	38365	96.8%

Minimum 10 values
Maximum 10 values

Value	Count	Frequency (%)
0	2	< 0.1%
259.85	1	< 0.1%
264.552	1	< 0.1%
264.57	4	< 0.1%
264.588	4	< 0.1%
264.606	7	< 0.1%
264.624	22	0.1%
264.642	17	< 0.1%
264.66	33	0.1%
264.678	36	0.1%

Value	Count	Frequency (%)
557.575	11	< 0.1%
335.063	1	< 0.1%
335.009	1	< 0.1%
334.882	1	< 0.1%
334.792	1	< 0.1%
334.683	1	< 0.1%
334.629	5	< 0.1%
334.484	1	< 0.1%
334.376	1	< 0.1%
334.104	1	< 0.1%

Pressure in
Real number (ℝ)

HIGH CORRELATION
SKEWED

Distinct	260
Distinct (%)	0.7%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%
Mean	2.5166765

Minimum	-5.92575
Maximum	2.54503
Zeros	2
Zeros (%)	< 0.1%
Negative	11
Negative (%)	< 0.1%
Memory size	619.3 KiB

Quantile statistics

Minimum	-5.92575
5-th percentile	2.50741
Q1	2.51212
median	2.52044
Q3	2.52351
95-th percentile	2.53165
Maximum	2.54503
Range	8.47078
Interquartile range (IQR)	0.01139

Descriptive statistics

Standard deviation	0.14200144
Coefficient of variation (CV)	0.056424191
Kurtosis	3469.8809
Mean	2.5166765
Median Absolute Deviation (MAD)	0.00687
Skewness	-58.604905
Sum	99753.507
Variance	0.020164408
Monotonicity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
2.52351	2036	5.1%
2.52333	1517	3.8%
2.51212	1449	3.7%
2.5123	1413	3.6%
2.52206	1300	3.3%
2.52315	1287	3.2%
2.51248	1238	3.1%
2.51989	1154	2.9%
2.52224	1133	2.9%
2.52007	1131	2.9%
Other values (250)	25979	65.5%

Minimum 10 values
Maximum 10 values

Value	Count	Frequency (%)
-5.92575	11	< 0.1%
0	2	< 0.1%
2.47106	1	< 0.1%
2.47848	1	< 0.1%
2.48011	1	< 0.1%
2.48029	1	< 0.1%
2.48119	1	< 0.1%
2.48282	1	< 0.1%
2.483	2	< 0.1%
2.48535	1	< 0.1%

Value	Count	Frequency (%)
2.54503	1	< 0.1%
2.54485	1	< 0.1%
2.54467	6	< 0.1%
2.54449	2	< 0.1%
2.54322	1	< 0.1%
2.54286	1	< 0.1%
2.54232	6	< 0.1%
2.54214	7	< 0.1%
2.54196	14	< 0.1%
2.54178	19	< 0.1%

Pressure out
Real number (ℝ)

HIGH CORRELATION
SKEWED

Distinct	343
Distinct (%)	0.9%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%
Mean	2.5187015

Minimum	0
Maximum	5.92575
Zeros	2
Zeros (%)	< 0.1%
Negative	0
Negative (%)	0.0%
Memory size	619.3 KiB

Quantile statistics

Minimum	0
5-th percentile	2.50615
Q1	2.51085
median	2.51935
Q3	2.52188
95-th percentile	2.53038
Maximum	5.92575
Range	5.92575
Interquartile range (IQR)	0.01103

Descriptive statistics

Standard deviation	0.060012078
Coefficient of variation (CV)	0.023826595
Kurtosis	3037.1368
Mean	2.5187015
Median Absolute Deviation (MAD)	0.00669
Skewness	47.05494
Sum	99833.77
Variance	0.0036014495
Monotonicity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
2.52188	1914	4.8%
2.51085	1791	4.5%
2.52206	1689	4.3%
2.51103	1548	3.9%
2.5208	1403	3.5%
2.51899	1293	3.3%
2.52224	1256	3.2%
2.5217	1185	3.0%
2.52062	1159	2.9%
2.51121	1152	2.9%
Other values (333)	25247	63.7%

Minimum 10 values
Maximum 10 values

Value	Count	Frequency (%)
0	2	< 0.1%
2.46727	1	< 0.1%
2.47143	1	< 0.1%
2.47305	1	< 0.1%
2.4745	1	< 0.1%
2.47902	1	< 0.1%
2.47938	1	< 0.1%
2.47956	3	< 0.1%
2.47975	1	< 0.1%
2.48155	1	< 0.1%

Value	Count	Frequency (%)
5.92575	11	< 0.1%
2.5633	1	< 0.1%
2.56311	2	< 0.1%
2.56275	1	< 0.1%
2.56221	1	< 0.1%
2.56185	1	< 0.1%
2.55968	1	< 0.1%
2.5595	1	< 0.1%
2.55896	1	< 0.1%
2.55877	1	< 0.1%

pump current
Real number (ℝ)

HIGH CORRELATION
SKEWED
ZEROS

Distinct	464
Distinct (%)	1.2%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%
Mean	15.868839

Minimum	0
Maximum	56183.1
Zeros	37477
Zeros (%)	94.6%
Negative	0
Negative (%)	0.0%
Memory size	619.3 KiB

Quantile statistics

Minimum	0
5-th percentile	0
Q1	0
median	0
Q3	0
95-th percentile	5.233 × 10^-5
Maximum	56183.1
Range	56183.1
Interquartile range (IQR)	0

Descriptive statistics

Standard deviation	935.84463
Coefficient of variation (CV)	58.973728
Kurtosis	3598.5184
Mean	15.868839
Median Absolute Deviation (MAD)	0
Skewness	60.001579
Sum	628993.19
Variance	875805.17
Monotonicity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
0	37477	94.6%
5.233 × 10^-5	1482	3.7%
0.00020931	43	0.1%
56183.1	11	< 0.1%
0.0102562	5	< 0.1%
0.00523278	5	< 0.1%
1.27345	4	< 0.1%
6.30074	4	< 0.1%
5.2916	4	< 0.1%
0.152588	4	< 0.1%
Other values (454)	598	1.5%

Minimum 10 values
Maximum 10 values

Value	Count	Frequency (%)
0	37477	94.6%
5.233 × 10^-5	1482	3.7%
0.00020931	43	0.1%
0.00047095	2	< 0.1%
0.00083725	3	< 0.1%
0.0018838	3	< 0.1%
0.00523278	5	< 0.1%
0.00633166	1	< 0.1%
0.00753521	2	< 0.1%
0.0088434	1	< 0.1%

Value	Count	Frequency (%)
56183.1	11	< 0.1%
837.245	1	< 0.1%
306.959	1	< 0.1%
252.576	1	< 0.1%
219.693	1	< 0.1%
182.398	1	< 0.1%
178.512	1	< 0.1%
174.093	1	< 0.1%
173.14	1	< 0.1%
164.865	1	< 0.1%

Human Counter
Real number (ℝ)

HIGH CORRELATION
ZEROS

Distinct	13
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%
Mean	7.8640412

Minimum	0
Maximum	15
Zeros	8333
Zeros (%)	21.0%
Negative	0
Negative (%)	0.0%
Memory size	619.3 KiB

Quantile statistics

Minimum	0
5-th percentile	0
Q1	3
median	7
Q3	15
95-th percentile	15
Maximum	15
Range	15
Interquartile range (IQR)	12

Descriptive statistics

Standard deviation	5.5183122
Coefficient of variation (CV)	0.70171456
Kurtosis	-1.2692098
Mean	7.8640412
Median Absolute Deviation (MAD)	7
Skewness	-0.024250063
Sum	311707
Variance	30.451769
Monotonicity	Not monotonic

Histogram with fixed size bins (bins=13)

Value	Count	Frequency (%)
15	11569	29.2%
6	9311	23.5%
0	8333	21.0%
9	5364	13.5%
10	2261	5.7%
3	1456	3.7%
7	858	2.2%
1	310	0.8%
4	122	0.3%
2	37	0.1%
Other values (3)	16	< 0.1%

Minimum 10 values
Maximum 10 values

Value	Count	Frequency (%)
0	8333	21.0%
1	310	0.8%
2	37	0.1%
3	1456	3.7%
4	122	0.3%
6	9311	23.5%
7	858	2.2%
8	1	< 0.1%
9	5364	13.5%
10	2261	5.7%

Value	Count	Frequency (%)
15	11569	29.2%
12	1	< 0.1%
11	14	< 0.1%
10	2261	5.7%
9	5364	13.5%
8	1	< 0.1%
7	858	2.2%
6	9311	23.5%
4	122	0.3%
3	1456	3.7%

temperature
Categorical

Distinct	2
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Memory size	619.3 KiB

74.2574	39635
0.0	2

Length

Max length	7
Median length	7
Mean length	6.9997982
Min length	3

Characters and Unicode

Total characters	277451
Distinct characters	6
Distinct categories	2 ?
Distinct scripts	1 ?
Distinct blocks	1 ?

The Unicode Standard assigns character properties to each code point, which can be used to analyse textual variables.

Unique

Unique	0 ?
Unique (%)	0.0%

Sample

1st row	74.2574
2nd row	74.2574
3rd row	74.2574
4th row	74.2574
5th row	74.2574

Common Values

Value	Count	Frequency (%)
74.2574	39635	> 99.9%
0.0	2	< 0.1%

Length

Histogram of lengths of the category

Common Values (Plot)

Value	Count	Frequency (%)
74.2574	39635	> 99.9%
0.0	2	< 0.1%

Most occurring characters

Value	Count	Frequency (%)
7	79270	28.6%
4	79270	28.6%
.	39637	14.3%
2	39635	14.3%
5	39635	14.3%
0	4	< 0.1%

Most occurring categories

Value	Count	Frequency (%)
Decimal Number	237814	85.7%
Other Punctuation	39637	14.3%

Most frequent character per category

Decimal Number

Value	Count	Frequency (%)
7	79270	33.3%
4	79270	33.3%
2	39635	16.7%
5	39635	16.7%
0	4	< 0.1%

Other Punctuation

Value	Count	Frequency (%)
.	39637	100.0%

Most occurring scripts

Value	Count	Frequency (%)
Common	277451	100.0%

Most frequent character per script

Common

Value	Count	Frequency (%)
7	79270	28.6%
4	79270	28.6%
.	39637	14.3%
2	39635	14.3%
5	39635	14.3%
0	4	< 0.1%

Most occurring blocks

Value	Count	Frequency (%)
ASCII	277451	100.0%

Most frequent character per block

ASCII

Value	Count	Frequency (%)
7	79270	28.6%
4	79270	28.6%
.	39637	14.3%
2	39635	14.3%
5	39635	14.3%
0	4	< 0.1%

water level
Categorical

Distinct	2
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Memory size	619.3 KiB

500.0	38835
800.0	802

Length

Max length	5
Median length	5
Mean length	5
Min length	5

Characters and Unicode

Total characters	198185
Distinct characters	4
Distinct categories	2 ?
Distinct scripts	1 ?
Distinct blocks	1 ?

The Unicode Standard assigns character properties to each code point, which can be used to analyse textual variables.

Unique

Unique	0 ?
Unique (%)	0.0%

Sample

1st row	500.0
2nd row	500.0
3rd row	500.0
4th row	500.0
5th row	500.0

Common Values

Value	Count	Frequency (%)
500.0	38835	98.0%
800.0	802	2.0%

Length

Histogram of lengths of the category

Common Values (Plot)

Value	Count	Frequency (%)
500.0	38835	98.0%
800.0	802	2.0%

Most occurring characters

Value	Count	Frequency (%)
0	118911	60.0%
.	39637	20.0%
5	38835	19.6%
8	802	0.4%

Most occurring categories

Value	Count	Frequency (%)
Decimal Number	158548	80.0%
Other Punctuation	39637	20.0%

Most frequent character per category

Decimal Number

Value	Count	Frequency (%)
0	118911	75.0%
5	38835	24.5%
8	802	0.5%

Other Punctuation

Value	Count	Frequency (%)
.	39637	100.0%

Most occurring scripts

Value	Count	Frequency (%)
Common	198185	100.0%

Most frequent character per script

Common

Value	Count	Frequency (%)
0	118911	60.0%
.	39637	20.0%
5	38835	19.6%
8	802	0.4%

Most occurring blocks

Value	Count	Frequency (%)
ASCII	198185	100.0%

Most frequent character per block

ASCII

Value	Count	Frequency (%)
0	118911	60.0%
.	39637	20.0%
5	38835	19.6%
8	802	0.4%

Auto

The auto setting is an interpretable pairwise column metric of the following mapping:

Variable_type-Variable_type : Method, Range
Categorical-Categorical : Cramer's V, [0,1]
Numerical-Categorical : Cramer's V, [0,1] (using a discretized numerical column)
Numerical-Numerical : Spearman's ρ, [-1,1]

The number of bins used in the discretization for the Numerical-Categorical column pair can be changed using config.correlations["auto"].n_bins. The number of bins affects the granularity of the association you wish to measure.

This configuration uses the recommended metric for each pair of columns.

Spearman's ρ

The Spearman's rank correlation coefficient (ρ) is a measure of monotonic correlation between two variables, and is therefore better in catching nonlinear monotonic correlations than Pearson's r. It's value lies between -1 and +1, -1 indicating total negative monotonic correlation, 0 indicating no monotonic correlation and 1 indicating total positive monotonic correlation.

To calculate ρ for two variables X and Y, one divides the covariance of the rank variables of X and Y by the product of their standard deviations.

Pearson's r

The Pearson's correlation coefficient (r) is a measure of linear correlation between two variables. It's value lies between -1 and +1, -1 indicating total negative linear correlation, 0 indicating no linear correlation and 1 indicating total positive linear correlation. Furthermore, r is invariant under separate changes in location and scale of the two variables, implying that for a linear function the angle to the x-axis does not affect r.

To calculate r for two variables X and Y, one divides the covariance of X and Y by the product of their standard deviations.

Kendall's τ

Similarly to Spearman's rank correlation coefficient, the Kendall rank correlation coefficient (τ) measures ordinal association between two variables. It's value lies between -1 and +1, -1 indicating total negative correlation, 0 indicating no correlation and 1 indicating total positive correlation.

To calculate τ for two variables X and Y, one determines the number of concordant and discordant pairs of observations. τ is given by the number of concordant pairs minus the discordant pairs divided by the total number of pairs.

Cramér's V (φc)

Cramér's V is an association measure for nominal random variables. The coefficient ranges from 0 to 1, with 0 indicating independence and 1 indicating perfect association. The empirical estimators used for Cramér's V have been proved to be biased, even for large samples. We use a bias-corrected measure that has been proposed by Bergsma in 2013 that can be found here.

Phik (φk)

Phik (φk) is a new and practical correlation coefficient that works consistently between categorical, ordinal and interval variables, captures non-linear dependency and reverts to the Pearson correlation coefficient in case of a bivariate normal input distribution. There is extensive documentation available here.

Count
Matrix

A simple visualization of nullity by column.

Nullity matrix is a data-dense display which lets you quickly visually pick out patterns in data completion.

First rows
Last rows

	salinity	turbidity	ORP	PH	TDS	Pressure in	Pressure out	pump current	Human Counter	temperature	water level
time_stamp
2022-11-01 11:01:16	287.18	9.51	774.05	7.42	287.18	2.54	2.53	0.00	0.00	74.26	500.00
2022-11-01 11:01:26	287.18	9.36	773.60	7.42	287.18	2.54	2.53	0.00	0.00	74.26	500.00
2022-11-01 11:01:36	287.12	9.36	773.87	7.42	287.12	2.54	2.53	0.00	0.00	74.26	500.00
2022-11-01 11:01:46	287.25	8.57	773.96	7.41	287.25	2.54	2.53	0.00	0.00	74.26	500.00
2022-11-01 11:01:57	287.18	8.89	773.60	7.42	287.16	2.54	2.53	0.00	0.00	74.26	500.00
2022-11-01 11:02:07	287.18	8.57	774.14	7.41	287.18	2.54	2.53	0.00	0.00	74.26	500.00
2022-11-01 11:02:17	287.21	9.04	774.14	7.41	287.21	2.54	2.53	0.00	0.00	74.26	500.00
2022-11-01 11:02:27	287.21	8.89	773.78	7.41	287.21	2.54	2.53	0.00	0.00	74.26	500.00
2022-11-01 11:02:37	287.16	8.73	773.69	7.41	287.16	2.54	2.53	0.00	0.00	74.26	500.00
2022-11-01 11:02:46	287.23	8.73	773.96	7.41	287.16	2.54	2.53	0.00	0.00	74.26	500.00

	salinity	turbidity	ORP	PH	TDS	Pressure in	Pressure out	pump current	Human Counter	temperature	water level
time_stamp
2022-11-07 18:58:35	288.13	32.99	722.60	7.59	288.13	2.52	2.51	0.00	0.00	74.26	500.00
2022-11-07 19:58:45	287.39	32.68	721.42	7.59	287.39	2.52	2.51	0.00	0.00	74.26	500.00
2022-11-07 19:58:55	287.46	32.84	721.33	7.59	287.46	2.52	2.51	0.00	0.00	74.26	500.00
2022-11-07 20:59:05	286.72	32.05	718.53	7.59	286.72	2.52	2.51	0.00	0.00	74.26	500.00
2022-11-07 20:59:15	286.74	31.74	718.62	7.58	286.74	2.52	2.51	0.00	0.00	74.26	500.00
2022-11-07 21:59:25	286.05	31.27	719.62	7.60	286.05	2.52	2.51	0.00	0.00	74.26	500.00
2022-11-07 21:59:35	286.02	31.11	719.43	7.59	286.02	2.52	2.51	0.00	0.00	74.26	500.00
2022-11-07 22:59:44	285.29	31.27	717.08	7.59	285.29	2.52	2.52	0.00	0.00	74.26	500.00
2022-11-07 22:59:54	285.26	30.96	717.36	7.59	285.26	2.52	2.51	0.00	0.00	74.26	500.00
2022-11-07 23:59:54	284.66	31.11	715.18	7.58	284.66	2.52	2.51	0.00	0.00	74.26	500.00

Most frequently occurring

	salinity	turbidity	ORP	PH	TDS	Pressure in	Pressure out	pump current	Human Counter	temperature	water level	# duplicates
9	557.58	-4375.26	3002.87	20.74	557.58	-5.93	5.93	56183.10	0.00	74.26	500.00	11
0	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	500.00	2
1	274.23	32.37	777.49	7.43	274.23	2.53	2.53	0.00	0.00	74.26	500.00	2
2	274.23	32.37	777.67	7.44	274.23	2.53	2.53	0.00	0.00	74.26	500.00	2
3	275.62	30.64	743.67	7.44	275.62	2.51	2.51	0.00	10.00	74.26	500.00	2
4	332.50	14.97	677.57	5.35	332.50	2.54	2.54	0.00	0.00	74.26	500.00	2
5	332.58	14.50	677.75	5.35	332.58	2.54	2.54	0.00	0.00	74.26	500.00	2
6	333.18	14.81	677.66	5.35	333.18	2.54	2.54	0.00	0.00	74.26	500.00	2
7	333.18	14.81	677.66	5.35	333.18	2.54	2.54	0.00	0.00	74.26	500.00	2
8	333.80	14.97	677.57	5.34	333.80	2.54	2.54	0.00	0.00	74.26	500.00	2

Overview

Variables

Common Values

Length

Common Values (Plot)

Most occurring characters

Most occurring categories

Most frequent character per category

Decimal Number

Other Punctuation

Most occurring scripts

Most frequent character per script

Common

Most occurring blocks

Most frequent character per block

ASCII

Common Values

Length

Common Values (Plot)

Most occurring characters

Most occurring categories

Most frequent character per category

Decimal Number

Other Punctuation

Most occurring scripts

Most frequent character per script

Common

Most occurring blocks

Most frequent character per block

ASCII

Interactions

Correlations

Auto

Spearman's ρ

Pearson's r

Kendall's τ

Cramér's V (φc)

Phik (φk)

Missing values

Sample

Duplicate rows

Most frequently occurring